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ABSTRACT 

This study aims to test the usability of MyLA (My Learning Analytics), an application for students at two German 
universities: The Cooperative State University Mannheim and University of Mannheim. The participating universities 
focus on the support of personalized and self-regulated learning. MyLA collects data such as learning behavior and 
strategies as well as personality traits. This paper presents the findings of a usability test of the web app prototype. A total 
of 105 students from both universities participated in the study. In addition to a quantitative usability survey, the app 
navigation and design was evaluated through an eye tracking investigation with seven participants. The findings indicate 
that the MyLA prototype is easy to use but requires slight modifications concerning the app design. 
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1. INTRODUCTION 

The utilization of technologies in everyday life is constantly growing. For students, technologies are mostly 
indispensable. For example, 95 % of 14 to 29 years old Germans used a smartphone in 2016 (Statista / 
Bitkom, 2017). Flowever, the potential of mobile devices has not been hilly developed in many universities. 
There are numerous possibilities how digital technologies can improve learning at higher education 
institutions. In the NMC Horizon Report 2017, six key trends were identified to adopt technologies in the 
higher education sector: (a) advancing cultures of learning (higher involvement in innovation development 
processes), (b) deeper learning approaches (connection of learning with the real world), (c) growing focus on 
measuring learning (analytics of data out of learning environments), (d) redesigning learning spaces 
(improvement of the technical infrastructure), (e) blended learning designs (combination of online and 
face-to-face learning), and (t) collaborative learning (social interaction and intercultural experiences) 
(Adams Becker et al., 2017). The presented study covers two emerging fields of research in higher education: 
(1) learning analytics and (2) mobile learning. 

Learning analytics (LA) use static and dynamic information to support students’ learning process and 
optimize learning environments. Besides its flexibility, the main advantages of LA are personalization and 
the real-time availability of data (Ifenthaler et al., 2014). Lecturers can use rich data for pedagogical 
decision-making, understand individual performance development of students, identity potential lack of 
students’ capabilities, or the need for curricular improvements (Mattingly et al., 2012). With LA, both 
students and lecturers can reflect on and improve their communication skills. By capturing, analyzing, and 
visualizing the available information about learning and teaching, lecturers are able to make more reliable 
predictions about their students’ academic success (Macfadyen and Dawson, 2012). Furthermore, students at 
risk can be identified and given support through personalized pedagogical interventions (Lockyer et al., 
2013). Successful applications of LA at universities are for example. Course Signals at Purdue University, 
which identifies students at risk using an approach similar to a traffic light system (green - no risk, yellow - 
potential risk, red - risky). Students and lecturers can identify needs for action to improve their learning 
situation. Furthermore, lecturers are able to intervene and help early (Ifenthaler and Schumacher, 2016). At 
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the University of Wollongong, SNAPP (Social Networks Adapting Pedagogical Practice) is used. The main 
purpose of this system is to increase collaborative learning. For example, conversations in forums are 
analyzed for investigating the relationship between students and excluded students can be determined. The 
system is also used for reflection at the end of a lecture (Sclater et al., 2016). 

Mobile Learning - m-learning - enables learning through personal portable electronic devices across 
multiple context. Through the usage of mobile devices students have access to learning materials more easily. 
The precondition is the availability of web-connected devices. In addition, students are independent from 
locations, time and they can communicate asynchronous (Lin et al., 2016). Mobile learning supports 
self-regulated learning on the one hand and it’s an essential element of blended learning environments on the 
other hand (Al Saleh and Bhat, 2015). An example for a German mobile application is ARSnova of the THM 
University of Applied Sciences GieBen. ARS stands for Audience Response System. Via this app, students 
and lecturers can communicate interactively. For example, students can ask questions anonymously during a 
lecture and the lecturer is able to answer the questions in near real-time. Moreover, the communication can 
take place before or after a lecture. Another function is the evaluation of a lecture directly in the app 
(ARSnova, 2015). 

The app MyLA (My Learning Analytics) of the Cooperative State University Mannheim and the 
University of Mannheim targets to improve learning processes at universities. MyLA provides ubiquitous 
communication in form of short messages from students to their lecturer and vice versa. This is especially 
useful for dual system courses where students are often away from the campus. Using this data, lecturers can 
adapt their lectures at university and implement personalized interventions, for example, adaption of learning 
materials or a detailed view at topics the students have problems with. Furthermore, the app supports a more 
personalized and individual way of learning. The students can document their learning motivation through 
their learning process. The app provides statistics, which enable students to observe their personal progress 
over time. In summary, MyLA combines learning analytics and mobile learning with the extension of 
personal learning. 

This paper focuses on the components of the MyLA app prototype and the usability testing of the app. 


2. MYLA APP 

The web app prototype of MyLA (My Learning Analytics) consists of three different main units 
(see Figure 1): (I) My Profile, (II) My Learning, and (III) My Progress. The general page structure consists of 
the following: 

• Header: with an icon for the side menu, the MyLA logo and an icon for the home button (except on 
the index page) 

• Main: with contents (e.g. select lists in the profile data or pinboard elements like button and post) 

• Footer: with the linked Cooperative State University Mannheim logo, imprint and privacy, icons 
linked to: contact, frequently asked questions (FAQ), settings and search function 
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Figure 1. Web App Prototype MyLA (translated from German) 

I. My Profile contains two subcategories: (1) profile data with input options like username or 
university and (2) the trophy center where app users have access to their rewards for i.e. when 
entering the profile data or participating on a certain questionnaire. This section includes 
administrative information of every app user. 

II. My Learning contains two subcategories: (1) the pinboard where students can create posts for 
their lecturers and (2) the survey center where students can response to regular conducted 
questionnaires. Within this section of the app, the students can communicate with their lecturers 
and vice versa. Via pinboard, students can post messages by using tags (e.g. ask questions or 
point to a problem). This will be only visible for the responsible lecturer on their dashboard 
interface. 

III. My Progress contains two subcategories: (1) MyLA data where students can enter personal data 
like learning motivation or learning effort and (2) MyLA stats where the MyLA data will be 
visualized in charts. This part displays the individual progress in additional (learning) factors. 

The design of the app was realized to accommodate the corporate design of the Cooperative State 
University Mannheim and the University of Mannheim. Therefore, one significant color of each education 
institute had been extracted. As a next step, the colors were combined and supplemented by neutral colors. 

According to the approach of learning analytics, the first prototype of MyLA app was designed to capture 
user data. Further steps will be collecting the data reports and deriving individual actions for students. 
Thereby, the main objective of developing personalized and adapted learning environments will be striven. 


3. USABILITY TESTING 
3.1 Research Questions 

The usability testing focused on three major research questions: 

1. How intuitive is the MyLA prototype (design, navigation) for students? 

2. Is there room for improvement for the development of the MyLA prototype? 

3. How can the empirical results (quantitative and qualitative) help to optimize the MyLA prototype 
for its initial implementation? 
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The questionnaire was conducted in order to evaluate design, navigation, text elements and used icons of 
MyLA. Beforehand, the students were shortly introduced to the topic of Learning Analytics. Afterwards, they 
were able to view the MyLA app either on web browser (Cooperative State University Mannheim) or on 
mobile browser (University of Mannheim) and afterwards responded to an online-questionnaire. For the 
purpose of comparability, none of the participants has used the MyLA web app before. 

3.2 Methodology 

3.2.1 Participants 

The usability test was conducted with 105 students (56 Cooperative State University Mannheim, 49 
University of Mannheim; 51 female, 54 male) in April 2017. The average age of the participants was 23.65 
years ( SD = 3.72, Min = 19, Max = 35). The majority of the respondents (N = 99) were studying in the field 
of business administration. More than half of the students were enrolled in a bachelor program (53.33 %) and 
46.67 % were studying in a master program. In addition, seven students (4 female, 3 male) took part in an 
additional eye tracking study. In average, participants reported that they are spending 29 days (M = 28.90, SD 
= 4.56) per month with apps generally, but only four days (M = 3.65, SD = 6.06) days per month with apps 
for learning. 

3.2.2 Design and Procedure 

The usability test was divided into two parts using a standardized instrument (see 3.2.3): First, the 
participants made themselves familiar with MyLA app prototype via web browser or mobile browser. This 
included the navigation through the app and reviewing the design and the app’s structure. Second, they 
responded to an online-questionnaire which was structured as follows: (1) Socio-demographic information, 
general usage of mobile devices and technologies, (2) MyLA-specific questions (open and closed questions). 
The MyLA-specific questions focused on navigation and navigation elements, design, and app structure. The 
main group (N = 98) followed this procedure. A smaller group (N = 7) participated in an eye tracking study 
(see 3.2.4). A significant difference between the participants was the device on which MyLA was tested: One 
group (Cooperative State University Mannheim) tested on a web browser via personal computer and another 
group (University of Mannheim) on a mobile browser via tablet. In order to ensure the anonymity various 
identification numbers had been given to the students. 

3.2.3 Usability Instrument 

The feedback of the students was committed via an online-questionnaire. The question pool within the 
MyLA-specific part was chosen following the usability testing instrument developed for FIIMATT (or Highly 
integrated model assessment technology and tools). The instrument has been successfully tested for reliability 
and validity (see Pirnay-Dummer et al., 2010). For the MyLA Usability Testing, 13 items had been chosen. 
An example: “I found it easy to navigate through the software ”, All items were answered on a five-point 
Likert scale ranging from highly agree (5) to highly disagree (1). Figure 2 shows the thirteen items used in 
the usability testing. 

3.2.4 Eye Tracking 

According to Rayner (2009), eye movements are connected to a participant's attention and can therefore 
contribute to the usability testing of screen-based applications. Eye tracking is a standard methodology to 
record the eye movement of participants. The data evaluation is conducted with special eye tracking 
software. A very useful feature is the report function that visualizes the eye movements through heat maps or 
gaze plots (Kurzhals et al., 2017). Common observations in eye tracking studies include (Ehmke and Wilson, 
2007): 

• Fixation points: where participants have a long look, 

• First look: where participants look first, or 

• Non-looking: elements participants do not pay attention to. 
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After a short introduction, the students were instructed to solve three tasks by using the app prototype. 
The difficulty of each task was ascending. Therefore, seven AOI (Areas Of Interests) were defined for the 
analysis with Tobii Pro Studio software. All participants were recorded with regard to their “first look” and 
on which MyLA-contents they looked more often (“fixation points”). Furthermore, the student’s solution 
approach was analyzed. The topics of the three tasks can be summarized as: 

1. Calling the Cooperative State University Mannheim website as soon as possible. 

2. Selecting and saving their respective university in the subcategory profile data. 

3. Creating a post for a lecturer. 

Students from the eye tracking study also participated in the survey-based usability test. 

3.3 Results 

The investigation of the results is divided into the survey-based and eye tracking usability test. Data analysis 
was conducted using IBM SPSS 23.0 and Tobii Pro Studio. 

3.3.1 Survey-based Usability Test 

The main part of the questionnaire was to investigate the app’s navigation and design. Figure 2 shows the 
findings of the thirteen items from the survey-based usability test. 

MyLA navigation and structure 


It is easy to learn how to work with MyLA app. 4.45 (sd . 83 ) 

All functions can be used without support. 4.25 (SD .76) 

I found it easy to navigate through the app. 4.36 (SD .82) 

The screen layout is clear and comprehensible. 4.25 (sd . 90 ) 

The navigation of the app is user-friendly. 4.28 (SD .85) 

The text layout is sensible, clear and easy to read. 4.23 (SD . 92 ) 


1 highly 5 highly 

disagree agree 

Figure 2. Bar Charts with the results of the MyLA-speciftc questions. Own Figure 

The bar charts in Figure 2 are divided into two sections. The chart on left side shows the results 
concerning the navigation and structure of the MyLA prototype. The second chart on right side highlights the 
outcomes regarding the design and colors of MyLA. According to navigation and structure it is conspicuous 
that all six average values are constantly high. The highest value was reported for the simplicity of MyLA 
app prototype with an average of 4.45 (SD = .83). Followed by “I found it easy to navigate through the app ” 
(M = 4.36, SD = .82) and “ The navigation of the app is user-friendly ” (M = 4.28, SD = .85). Based on the 
second chart it is obvious that there were some divergent opinions concerning the design and colors of 
MyLA. The lowest rated value was the use of color with an average of 3.28 (SD = 1.15). In addition, the 
participants ranked the design of MyLA as “ optically appealing ” with 3.36 (SD = 1.19). 


MyLA design and colors 


The quality of the graphics is good, i.e. they have clear 
lines, forms, and contrast and are well-designed. 

The design of the app is optically 


3.97 (sd . 91 ) 


appealing. 


3.36 (sd 1.19) 


The quality of the colors is good, e.g. clear , eo 
contrast. 3.68 ( sdi.13) 


The colors are a useful aid for explaining 
how to operate the app. 


3.43 (sd 1 . 10 ) 


In general, the use of color is effective, 
sensible and motivating. 


3.28 (sd 1 . 15 ) 


The colors used to simplify the operation of the 


3.86 (sd 1 . 06 ) 


app are applied uniformly. 

The graphics are comprehensible designed. 3.98 (sd . 99 ) 


1 highly 
disagree 


5 highly 
agree 
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3.3.2 Eye Tracking Usability Test 

For the purpose of statistical analysis either the time of first fixation or the time to first mouse click were 
calculated. Table 1 shows the average times the pilot tester (conducted by a research team member) and the 
participants needed to complete the three tasks. The eye tracking study was implemented using the web 
browser version of MyLA (on a personal computer). 

Table 1. Overview of the eye tracking usability test (all values in seconds) 


Tasks _ 

01 - Website challenge 
(click on a logo) 


Pilot-Test (N = 

1.91 


Participants (N = 7) 

M = 4.86, SD = 2.46 
(solved by 5 of 7) 


02 - University selection 10.09 M = 14.55, SD = 9.43 

(in the profile) (solved by 5 of 7) 


03 - Pinboard post 4.81 M= 10.12, SD = 9.02 

(create a new one) _ (solved by 7 of 7) 


Table 1 shows that the majority of the participants (at least 5) were able to solve the simulated tasks. For 
task three, the students needed more time to find the solution in comparison to the first two tasks when 
compared to the benchmark of the pilot test. A possible explanation may be an unclear description of the 
task. Some students were not able to find the pinboard on a direct way (via chapter icon “My Learning”). 
Flowever, all participants completed task three through an alternative solution (via side navigation menu). 

As a next step, a heat map analysis was conducted which is a reflection of the screen where the 
participants viewed longer than other parts (Ehmke and Wilson, 2007) identifying gaze behavior precisely 
(Duchowski et al., 2012). Through the accumulation of all single viewpoints of a participant, the fixation 
points can be highlighted. Figure 3 shows the accumulation of the fixation points recorded by all participants. 
The viewpoints were predominantly recorded on the left side of MyLA prototype. For solving task one 
(pictured in the heat map of Figure 3), the participants had to look at the left side in order to find the 
Cooperative State University Mannheim logo. Additionally, it has to be considered that the participants saw 
the app via a desktop browser, hence, the screen width was obviously wider than on a mobile device. 



Figure 3. Heat map showing the cumulated fixation points of all Eye Tracking participants. 
Heat map exported using Tobii software 
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4. DISCUSSION AND CONCLUSION 

Usability testing is a very useful and advantageous method for formative evaluation of the development 
process. The project MyLA can benefit from the valuable input of the potential app user. 

With regard to research question 1 (“How intuitive is the MyLA prototype (design, navigation) for 
students?”), it can be suggested that the prototype is intuitive for the target group. Overall, the results were 
predominantly positive (all average values were higher than 3), however, some issues were identified for 
improvement, especially with regard to colors. 

In addition, students provided feedback within the scope of open questions. The students evaluated the 
app’s clarity, simplicity, navigation/structure and features as very positive. Regarding the assessment of the 
color scheme the responses were heterogeneous. Critical issues were partly used dark colors as well as color 
combinations. Moreover, some students mentioned that the app contained a broad color spectrum. 
Furthermore, they suggested additional features for future app versions, for example, a calendar function. 
With regard to research question 2 (“Is there room for improvement for the development of MyLA 
prototype?”), it can be summarized that the colors need to be adjusted. 

To answer the research question 3 (“How can the empirical results (quantitative and qualitative) help to 
optimize the MyLA prototype for the main survey?”), the following aspects can be summarized: The 
involvement of the target group (students) was very important at this early project stage. The reason for that 
is very simple, because the students are prospective users of MyLA. Therefore, it is inevitable to get them 
highly involved. The success of a project depends on its acceptance. If the acceptance is high, the potential 
usage can be high, too. For reaching a large consumption of MyLA it is necessary to implement the students’ 
feedback and recommendations. With help of these new insights future adaptions can be managed. Hence, 
the findings of the MyLA usability testing provided detailed insights to optimize the app prototype. Some 
lessons learned of the MyLA usability testing were the following proven statements: 

• The handling of the MyLA web app prototype is intuitive. 

• The app’s structure is easy to learn. 

• The navigation within the app is clear and user-friendly. 

• The students mostly like the idea of MyLA. 

• The design and colors can be improved, because the opinions deviate fairly high. 

The next steps in the project will be the further development and implementation of the app prototype 
regarding the panel study in autumn 2017 Therefore, the technical infrastructure has to be developed. 
Furthermore, the feedback of the students concerning navigation and design has to be verified and adapted in 
the app. Additionally, the dashboard for lecturers, as associated software will be finalized. For generating a 
high acceptance within the project, the researchers want to involve the target group (students). By regularly 
viewing new features and other technical changes they make sure that the application will meet the students’ 
needs. 

Currently, the field implementation of the MyLA app is being prepared. Within this next project stage, 
more data will be available about the target group and the use of the application. According to the literature, 
the research area of learning analytics is still in its infancy in German-speaking countries (Ifenthaler and 
Schumacher, 2016). Moreover, with the advance of digital technologies, new skills and competencies are 
required from students at higher education institutions as well as their future workplace. In addition, the 
situation of non-traditional academics will be considered within the project. The combination of learning 
analytics and mobile learning seems to be beneficial in order to contribute to the individual demands of a 
diverse group of learners. For example, the project team wants to identify patterns of learning behavior and 
the management of learning tasks given a high demand of workload at the university and the workplace. In 
summary, the project can be seen as an important contribution to the topic of learning analytics for higher 
education in Germany. 
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